Skip to content

Force HTTP/1.1 in pubchem_get to dodge buggy h2 client at the compat floor - #83

Draft
ChrisRackauckas-Claude wants to merge 5 commits into
masterfrom
agent/82-httptimeout
Draft

ChrisRackauckas-Claude wants to merge 5 commits into
masterfrom
agent/82-httptimeout

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Filed by an AI agent running as @ChrisRackauckas; Chris has not seen this.
Harness: Devin CLI 3000.11.228 · Model: swe-2-high
Conversation: local job 82-devin (no session URL)

Please ignore this PR until it has been reviewed by @ChrisRackauckas.

What changed and why

pubchem_get in src/data.jl now passes protocol = :h1 to HTTP.get, so PubChem/Rhea requests use HTTP/1.1 instead of HTTP.jl's HTTP/2 client. At this package's compat floor (HTTP 2.3.1, what the Downgrade lane pins), the h2 client never returns connection-level receive-window credit for response bytes that arrive but are never read — so one throttle-induced mid-body failure permanently stalls every later request on the pooled connection, which is the deterministic http timeout during request (elapsed ~120 s) in test/glycolysis.jl (upstream: JuliaWeb/HTTP.jl#1371, fixed in HTTP 2.12). These are sequential REST GETs, so h2 multiplexing gains nothing anyway.

Closes #82

Also on the branch, as a separate commit: .factory-pre-push.json declaring Runic for .jl files — the worker pre-push hook requires an explicit declaration since this repo uses the SciML shared runic workflow rather than fredrikekre/runic-action directly (same file sibling checkouts carry).

Diagnosis summary

  • The Downgrade lane runs julia-downgrade-compat in deps mode on Julia lts (1.10.12), pinning HTTP 2.3.1.
  • CI stack shows the stall inside HTTP/2 body read (_wait_h2_body_progress! → i/o timeout) while fetching the large PUG-View JSON at src/data.jl:89, reached from glycolysis.jl:32 → species_from_cid.
  • PubChem throttles: a 150-request burst from here drew 104 StatusErrors, and ~20 s paced pauses occur on both HTTP 2.3.1 and 2.13.1.
  • On HTTP < 2.12 each abandoned/reset/padded body permanently shrinks the shared 64 KiB connection window. Once it hits zero, every later response stalls — and each retry lands on the same dead pooled connection, which is why the existing backoff did not help. This also explains why only the oldest-deps lane fails: current deps run HTTP ≥ 2.12, where the credit is returned.

Verification

Lane replica: downgrade.jl produced the floor Manifest.toml (HTTP 2.3.1); installed and run on julia +1.10 = 1.10.12, same as the lane.

Fail before (master src/data.jl):

$ julia +1.10 --project=. test/glycolysis.jl
ERROR: LoadError: http timeout during request (elapsed 120080084994 ns)
 ...
 [10] get_json_and_view @ src/data.jl:89
 [14] species_from_cid @ src/species.jl:150
 [21] pathway_reaction_to_reaction @ src/pathway.jl:123
 [28] top-level scope @ test/glycolysis.jl:32

Same error type, same frame stack, same ~120 s elapsed as CI (120151175323 ns there).

Pass after (same manifest, with protocol = :h1):

$ julia +1.10 --project=. -e 'using PubChemReactions; ...; include("test/glycolysis.jl")'
HTTP 2.3.1
GLYCOLYSIS PASS after fix in 180 s

Mechanism reproducer on HTTP 2.3.1's shared pooled client: one response body abandoned mid-read, then every later request on the pool stalled at read_idle_timeout (6 consecutive HTTP.TimeoutErrors at 20.2 s each — the CI signature), while protocol = :h1 returned the 3.4 MB record in 0.4 s on the same poisoned process state.

Existing tests: test/http_retry.jl passes with the fix under both HTTP 2.3.1 and 2.13.1. runic made no changes; typos is clean.

Full lane group on the same downgraded manifest (GROUP=Core julia +1.10 --project=. -e 'using Pkg; Pkg.test(allow_reresolve=false)'):

Test Summary:      | Pass  Total     Time
Core/glycolysis.jl |    5      5  4m45.4s
Test Summary: | Pass  Total  Time
Core/graph.jl |    3      3  4.4s
Test Summary:      | Pass  Total  Time
Core/http_retry.jl |    4      4  18.3s
Test Summary:   | Pass  Total     Time
Core/pathway.jl |   11     11  1m51.8s
Test Summary:   | Pass  Total  Time
Core/species.jl |   10     10  2.7s
     Testing PubChemReactions tests passed

Round-1 counted-review follow-up

  • Docstring version claim corrected: the h2 receive-window credit refund is absent at the 2.3.1 floor (verified source-level through 2.6.7) and present by 2.7.1 — previously overstated as "before 2.12".
  • Deterministic protocol-pin test: no clean seam was found and none was faked. Attempted, with real outcomes: (a) plaintext loopback — :auto already picks h1, cannot discriminate (same reason test/http_retry.jl never could); (b) local TLS server advertising ["h2","http/1.1"] with server-side Request.version capture — on master, pubchem_get's :auto attempt still lands h1 (h2 attempt fails over before the handler), so the observed version is identical with and without the patch; (c) trace/RequestEvent capture of the attempted protocol — control HTTP.get emits :h2 but master's pubchem_get emitted only :h1 events on this endpoint, so the test passes unfixed (reason for the h1 choice at that call site was not root-caused); (d) h2-only ALPN endpoint — ALPN mismatch is not fatal, the h1 request still completes. The behavior fix itself remains covered by the lane-faithful fail-before/pass-after above.

Note on the x86 Core lane (pre-existing upstream breakage)

The Core (julia 1, ubuntu-latest, x86) lane fails on this PR with LoadError: InexactError: convert(Int32, 0xffffffff). This is pre-existing on master and unrelated to this change: the error is raised inside HTTP.jl's own module include — const _HPACK_MAX_INTEGER = Int(typemax(UInt32)) at hpack.jl:747 overflows Int32 while HTTP.jl precompiles, before any PubChemReactions code runs. protocol = :h1 is a request-time keyword; it cannot affect HTTP.jl's own load. Reproducer (32-bit Julia, no package code involved):

$ julia +1.13.1~x86 -e 'using HTTP'   # HTTP v2.13.1
ERROR: LoadError: InexactError: convert(Int32, 0xffffffff)
  in expression starting at .../HTTP/src/hpack.jl:747
  in expression starting at .../HTTP/src/HTTP.jl:1

The constant is absent in HTTP 2.3.1, which is why the same PR's Downgrade lane (floor manifest) is green while the latest-deps x86 lane is red; master's x86 lane simply has not run since the HTTP release that introduced it. Fixing it requires an HTTP.jl-side change (e.g. Int64/UInt32 for _HPACK_MAX_INTEGER), which is out of scope here per the open x86-lane decision.

Not verified

  • The exact CI seeding event (503-retry abandon vs. server RST vs. DATA padding) is inferred from the HTTP.jl changelog plus observed throttling, not packet-captured.
  • GROUP=QA not run locally — the diff is one request kwarg plus a docstring paragraph.
  • get_page's Downloads.download path (libcurl) is unaffected by this change and not covered by the evidence.

What to push back on

  • This is a workaround for an upstream bug rather than a compat bump; the issue brief forbids pinning HTTP upward to dodge the downgrade, but raising HTTP = "2.12" is the alternative if a reviewer prefers it.
  • Whether forcing h1 costs anything a user could rely on: h1 keep-alive pooling still applies, and no PubChem/Rhea feature requires h2.

Risk assessment

  • Risk: low — one request kwarg on the package's internal fetch helper; no public API change.
  • Blast radius: every PubChem/Rhea GET in the package (pubchem_get is the single entry point); current-dep users also switch to h1.
  • Evidence: lane-faithful fail-before/pass-after on Julia 1.10.12 + HTTP 2.3.1, matching CI error type, frames, and elapsed time; upstream fix identified in the HTTP.jl changelog.
  • Independent review: pending fleet review.

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com


🤖 Posted by an AI agent — harness: Devin CLI 3000.11.228 · model: swe-2-high
Conversation: local job 82-devin (no session URL)

🤖 Automated review section from an AI agent running as Chris Rackauckas's account — not written or reviewed by Chris.

Fix by Muse CLI Code muse-wonder-full-1.0 (head f255f13)

No code change and no new commit: the failing Core lane is pre-existing upstream breakage, now fixed upstream, so this branch needs only a fresh CI run.

What was checked:

  • The only failing check on this PR is Core (julia 1, ubuntu-latest, x86), failing while HTTP.jl itself loads: LoadError: InexactError: convert(Int32, 0xffffffff) at HTTP/src/hpack.jl:747, before any PubChemReactions code runs. The line const _HPACK_MAX_INTEGER = Int(typemax(UInt32)) overflows Int on 32-bit Julia. A request-time keyword in pubchem_get cannot affect HTTP.jl's own precompilation.
  • Identical failure on an unrelated master-based PR from the same day, PR #84, run 40985865850, job 123836707175: same InexactError, same hpack.jl:747. All non-x86 lanes passed there too. The default branch has no CI run since 2026-10-17, which is why its x86 lane still shows green from before the breaking HTTP.jl release.
  • Broken range: the constant is present in HTTP.jl v2.12.0 through v2.15.0, and absent before v2.12.0. Upstream fixed it in v2.15.1 (2026-11-11): the line is now Int(min(typemax(UInt32), typemax(Int))). See the HTTP.jl v2.15.1 release notes and the closed upstream issue JuliaWeb/HTTP.jl#1558. This package's compat HTTP = "2.3.1" already admits v2.15.1 and later, so a fresh resolve selects a fixed HTTP.jl with no change here.
  • Excluding the broken middle range in [compat] was considered and rejected: compat entries are not architecture-specific, so it would block 64-bit users pinned to HTTP.jl 2.12–2.15.0 by other packages for a 32-bit-only upstream load bug that is already fixed.

Validation commands and observed output:

$ julia -e 'println(convert(Int32, typemax(UInt32)))'
ERROR: InexactError: convert(Int32, 0xffffffff)

$ GROUP=Core JULIA_NUM_THREADS=4 timeout 3600 julia --project=. -e 'using Pkg; Pkg.test()'
  [cd3eb016] + HTTP v2.15.2
Test Summary:      | Pass  Total     Time
Core/glycolysis.jl |    5      5  1m48.7s
Test Summary: | Pass  Total  Time
Core/graph.jl |    3      3  3.2s
Test Summary:      | Pass  Total   Time
Core/http_retry.jl |    4      4  15.7s
Test Summary:   | Pass  Total     Time
Core/pathway.jl |   11     11  1m19.8s
Test Summary:   | Pass  Total  Time
Core/species.jl |   10     10  7.0s
     Testing PubChemReactions tests passed

Not verified: the 32-bit x86 lane itself was not run locally (this machine has no 32-bit runtime libraries), and GROUP=QA was not run because there is no code change. The x86 lane should pass on the next CI run, which resolves HTTP.jl v2.15.2.

Next counted review must come from a smart model in a fresh session, not this fix session (risk.md § Independent reviews).


🤖 Posted by an AI agent — harness: Muse CLI Code · model: muse-wonder-full-1.0
Conversation: local fixer job SciML__PubChemReactions.jl__83 (no session URL)

🤖 Automated review section from an AI agent running as Chris Rackauckas's account — not written or reviewed by Chris.

Fix by Muse CLI Code muse-wonder-full-1.0 (head c8930da)

What changed:

  • Fetched upstream/master and merged it into agent/82-httptimeout. The merge was already up to date: the branch already contained upstream/master at 2c9fb53, so there were no new default-branch commits to bring in and no code change resulted.
  • Pushed one empty retrigger commit (c8930da) so CI re-resolves dependencies on the current registry. The only failing check was Core (julia 1, ubuntu-latest, x86), which failed inside HTTP.jl itself while loading, before any package code ran.
  • Diagnosis of that failure, re-verified locally on 32-bit Julia 1.12.7: HTTP 2.13.5 (the version the failing CI run resolved) fails to precompile with LoadError: InexactError: convert(Int32, 0xffffffff) at hpack.jl:747. HTTP 2.15.2 (what a fresh resolution now picks; 32-bit support was restored in HTTP 2.15.1) loads cleanly on the same 32-bit Julia. The failure is therefore pre-existing upstream breakage in a dependency, not caused by this PR, and no package code was changed for it.

Validation, run locally before pushing:

$ julia --project=@runic -m Runic --inplace src/data.jl
runic:0 (no changes)

$ typos src/data.jl
typos:0

$ GROUP=QA julia --project=. -e 'using Pkg; Pkg.test()'
  [cd3eb016] HTTP v2.15.2
Test Summary: | Pass  Total     Time
QA/qa.jl      |   21     21  3m10.0s
     Testing PubChemReactions tests passed

$ GROUP=Core julia --project=. -e 'using Pkg; Pkg.test()'
  [cd3eb016] HTTP v2.15.2
Test Summary:      | Pass  Total     Time
Core/glycolysis.jl |    5      5  4m54.8s
Test Summary: | Pass  Total  Time
Core/graph.jl |    3      3  4.5s
Test Summary:      | Pass  Total   Time
Core/http_retry.jl |    4      4  20.2s
Test Summary:   | Pass  Total     Time
Core/pathway.jl |   11     11  2m14.9s
Test Summary:   | Pass  Total  Time
Core/species.jl |   10     10  3.9s
     Testing PubChemReactions tests passed
CORE_EXIT:0

32-bit HTTP load check, run locally on Julia 1.12.7 (word size 32), fail before / pass after in dependency version only:

HTTP 2.13.5: ERROR: LoadError: InexactError: convert(Int32, 0xffffffff) at hpack.jl:747
HTTP 2.15.2: julia 1.12.7 wordsize 32 / HTTP loaded OK / HTTP version 2.15.2

Not verified: the full Core group on a 32-bit Julia was not run locally; the 32-bit evidence above covers the exact load failure that made the lane red, and the full Core group passed locally on 64-bit Julia 1.12. CI on the new head is the check for the 32-bit lane end to end.

Next counted review must come from a smart model in a fresh session, not this fix session (risk.md § Independent reviews).


🤖 Posted by an AI agent — harness: Muse CLI Code · model: muse-wonder-full-1.0
Conversation: local fixer job SciML__PubChemReactions.jl__83 (no session URL)

ChrisRackauckas and others added 2 commits October 28, 2026 18:46
…floor

HTTP.jl before 2.12 never returns connection-level HTTP/2 receive-window
credit for response bytes that arrive but are never read (abandoned bodies,
resets, padding). On the downgrade lane's HTTP 2.3.1 floor, a single
throttle-induced mid-body failure permanently drains the pooled connection's
64 KiB window, after which every later request stalls until the 120 s
read-idle timeout fires — the deterministic `http timeout during request`
error killing Core/glycolysis.jl (issue #82). Sequential REST GETs gain
nothing from h2 multiplexing, so request h1 explicitly.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI 3000.11.228
Agent-Model: swe-2-high
Agent-Session: local job 82-devin (no session URL)
The worker pre-push hook requires an explicit formatter declaration when the
repo does not call fredrikekre/runic-action directly (this repo uses the SciML
shared runic workflow), matching sibling job checkouts.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI 3000.11.228
Agent-Model: swe-2-high
Agent-Session: local job 82-devin (no session URL)
ChrisRackauckas and others added 3 commits October 29, 2026 01:21
Counted review flagged that HTTP 2.11.0 already returns connection-level
h2 window credit for unread body bytes. Verified at source: the refund
helper is absent through 2.6.7 and present by 2.7.1; the floor version
2.3.1 is demonstrably affected.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI 3000.11.228
Agent-Model: swe-2-high
Agent-Session: local job 82-devin (no session URL)
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Agent-Harness: Claude Code (claude-local)
Agent-Model: qwen3.8-flash-next
Agent-Session: local session e67c8f95-4e9d-4581-af70-717dfb1de194 (no session URL)
The branch already contains upstream/master (2c9fb53); merging it is a
no-op. The failing 32-bit Core lane resolved HTTP 2.13.5, which does not
load on 32-bit Julia. Current resolution picks HTTP 2.15.2, which loads
on 32-bit (32-bit support restored in HTTP 2.15.1). No code change.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Muse <noreply@meta.com>
Agent-Harness: Muse CLI Code
Agent-Model: muse-wonder-full-1.0
Agent-Session: local fixer job SciML__PubChemReactions.jl__83 (no session URL)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Downgrade Tests: deterministic 120 s HTTP timeout in reaction-data download errors Core/glycolysis.jl

2 participants